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Background 


•  Emergence  of  new  buzzwords  in  software  development 

-  Competitive  pressures  of  the  1990s  forced  software  companies  to  re¬ 
examine  their  deveiopment  processes  and  adopt  radicai  approaches.  As 
a  resuit,  the  industry  has  been  fiooded  with  buzzwords  associated  with 
“new”  deveiopment  approaches,  e.g.,  “internet  time”,  “extreme”,  and 
“agiie” 

•  A  flood  of  more  management  buzzwords  over  the  past  30  years. . . 

-  There  has  been  a  “bandwagon  effect”  of  popuiar  management 
movements  such  as  Totai  Quaiity  Management,  management  by 
objectives,  reinventing  government,  reengineering,  the  baianced 
scorecard.  Lean,  and  Six  Sigma 

•  However,  companies  that  have  been  earlier  identified  as  excellent  on 
the  basis  of  these  practices  later  turned  out  to  be  mediocre  or  outright 
failure  [Paparone  2009] 

-  Attempts  have  been  made  to  bring  agiiity  into  the  Air  Force  acquisition 
process  as  weii  [Evans  2003] 

•  Unfortunately,  the  Agile  Acquisition  initiative  did  not  gain  any  traction 

•  Consequently,  the  recent  recommendation  to  Pentagon  Brass:  “Stay 
Away  from  Management  Bestsellers...”  [Erwin  2009] 
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Motivation 


•  Despite  of  Erwin’s  recommendation . . . 

-  Agility  seems  to  be  a  simple  concept  and  it  is  commonly 
perceived  as  a  virtue 

-  Agile  methods  are  making  inroads  into  software  development 

•  Consequently,  the  idea  of  making  acquisition  agile  deserves 
a  closer  view 
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What  is  Agility? 


•  The  narrow  dictionary  definition: 

-  Ready  ability  to  move  with  quick  easy  grace 

-  Agile  is  being  quick  and  resourceful 

•  Agility  is  perceived  a  virtue 

•  In  business,  agility  is  considered  an  important  organizational  capability 

•  Unfortunately,  in  most  contexts  it  is  ill-defined  or  inconsistent 

-  Agility  does  not  simply  equate  with  speed,  as  the  following  examples  show 

•  Agility  conflicts  with  speed 

-  The  Titanic's  ability  to  turn  sharply  is  far  more  likely  to  avert  disaster 
than  increasing  its  top  speed  charging  straight  ahead 

•  Agility  requires  speed  but  also  requires  balance 

-  Martial  arts 

-  “Lean”  does  not  always  equate  with  “agile” 

•  Applying  “Lean”  might  increase  the  rigidity  of  a  process 

-  Rigidity  results  from  constraining  the  process  in  order  to  optimize 
the  case  “right  now” 
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Agility  in  Different  Contexts 


•  Agility  in  Software  Development 

-  The  prevailing  characteristic  of  an  agile  software  development 
organization  is  that  it  institutionally  embraces  the  agile  values  of  the 
Agile  Manifesto  [Agile  2001]: 

•  Individuals  and  interactions  over  processes  and  tools 

•  Working  software  over  comprehensive  documentation 

•  Customer  coiiaboration  over  contract  negotiation 

•  Responding  to  change  over  following  a  plan 

•  Agility  in  Business 

-  Authors  try  to  resolve  the  earlier  discussed  confusion  by  considering 
agility  as  a  two-dimensional  factor  [Masini  2005] : 

•  Range  Agiiity  -  speed 

•  Time  Agiiity  -  reaction 

-  They  concluded  that  when  it  comes  to  agility,  more  is  not  always 
better;  the  benefits  are  bounded  and  are  contingent  on  ease  and 
market  dynamics 
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Agility  in  Defense 


•  Agility  in  Defense 

-  There  is  a  confusion  about  the  need  for  systems  enabiing  war¬ 
fighter  agility  vs.  the  need  for  agile  acquisition  of  weapon 
systems 

•  There  is  no  argument  about  the  value  of  war-fighter  agility. 

However, 

-  War-fighter  agiiity  can  be  primariiy  supported  via 
weapons  design  and  fiexibie  architecture 

-  Faster  access  to  new  weapons  is  not  aiways  the  right 
soiution 

-  Fast  procurement  of  estabiished  weapons  is  important, 
but  out  of  scope  for  this  discussion 

•  Agility  in  Defense  Acquisition 

-  This  is  the  topic  of  this  presentation 
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Approach 


•  The  defense  acquisition  system  is  frequently  reformed 

-  The  Defense  Acquisition  Performance  Assessment  (DAPA)  of 
2006  mentions  9  major,  prior  acquisition  reforms,  DAPA  itself  is 
the  10^^,  and  the  most  recent.  Weapon  Systems  Acquisition 
Reform  Act  (WSARA)  of  2009  is  the 

-  it  is  worthwhile  to  pause  and  see  if  the  recommendations  call  for 
agility 

•  These  reforms  are  always  based  on  identified  problems  or 
outright  failures  of  the  acquisition  system 

-  Conversely  it  is  also  interesting  to  explore  if  agile  ideas,  e.g., 
Agile  Software  Development  or  the  Air  Force’s  Agile  Acquisition 
Initiative,  are  appropriate  to  mitigate  the  identified  problems  or 
failures 
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Context:  The  Acquisition  System  (Big  “A”  Acquisition  Process) 
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Perceived  to  be  a  simple  construct  of  three,  well  Integrated  inter-dependent  processes 
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Agile  Software  Development  for  New  Weapon  Systems 


Agile  Development  affects  only  the  smaller  context  of  the  DOD  5000.02  process 
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Examining  Agile  Software  Development  Values 


•  Individuals  and  interactions  over  processes  and  tools 

-  This  value  and  the  associated  practices  work  well  in  limited  settings  but  do 
not  scale  up 

•  E.g.,  the  following  numbers  represent  space  system  development 

-  Space  Vehicle  Software  is  embedded,  very  large 

•  Typical  size  512  Thousand  Delivered  Source  Instructions 
(KDSI),  including  bus  software  and  payload(s) 

-  Ground  Systems  are  even  larger 

•  Space  Shuttle  size  2,000  KDSI 

•  Satellite  control  systems  ~  4,700  KDSI 


The  role  of  tools  and  processes  is  very  critical  in  the 
development  of  such  large,  mission-critical  systems 
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Examining  Agile  Software  Development  Values  (cent.) 


•  Individuals  and  interactions  over  processes  and  tools  (cont.) 

-  The  following  data  shows  the  volatility  of  the  work  force* 

•  Government  Sector 

-  Average  Annual  Separation  Rate  17.4%  oftotai  Government 
Sector  empioyment 

-  Average  Annual  Hires  Rate  18.6% 

•  Information  Industry 

-  Average  Annual  Separation  Rate  38.3%  oftotai  Information 
Industry  employment 

-  Average  Annual  Hires  Rate  34% 

In  such  a  volatile  environment  development  cannot 
simply  depend  on  individuals  and  interactions 

•  Working  software  over  comprehensive  documentation 

Based  on  all  the  previous  arguments,  this  is  not  a 

realistic  principle  either 


*  Source:  Bureau  of  Labor  Statisticai  Database  for  the  period  of  2001  -  2008 
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Examining  Agile  Software  Development  Values  (cent.) 


•  Customer  collaboration  over  contract  negotiation 

-  The  Developer  is  far  removed  from  the  Actual  User  and  Actual 
Customer  (see  next  slide) 

•  Also,  there  is  a  substantial  tension  between  the  numerous 
stakeholders 


At  the  given  scale,  maintaining  constant  collaboration 
even  with  the  surrogate  customers  is  very  difficult 

•  Responding  to  change  over  following  a  plan 

-  The  mentioned,  typical  space  vehicle  software  development  of 
512  KDSI  would  require  roughly  a  6,420  Man-Month  effort, 
spreading  over  41  months,  involving  ^157  Full-time 
Equivalent  Software  Personnel 

At  the  given  scale,  maintaining  internal  collaboration 
without  coordinated  plans  is  not  feasible  either 
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Key  Stakeholders  in  the  Big  “A”  Acquisition  Process 


Legend: 

JCIDS  Joint  Capabilities  Integration  &  Development  System 
JROC  Joint  Requirements  Oversight  Council 
PPBE  Planning,  Programming,  Budgeting  &  Execution 
R  Performance  &  "Time  to  Need"  Requirements 

$  Allocated  Funding  for  Contracts 

$2  Allocated  Funding  for  Acquisition  Workforce 

There  is  a  tension  between  the  numerous  stakeholders  due  to  different  motivation/behavior 


•  The  process  elements  themselves  are  complex  and  ambiguous 

•  Process  integration  is  not  efficient  and  the  overall  system  is  unstable 
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Final  Thoughts  on  Agile  Software  Development 


•  Agile  ideas  using  the  rugby  metaphor  for  product  development  emerged 
as  early  as  1986  from  H.  Takeuchi  and  I.  Nonaka  [Takeuchi  1986]: 

-  Top  management  offers  only  a  general  strategic  direction  and  chalienging 
goais  (Does  not  provide  a  specific  work  pian) 

-  Hand-picked  but  seif-organizing  multi-discipiinary  teams 

-  Members  work  together  from  start  to  finish 

-  Process  is  characterized  by  constant  interaction  (“cross-fertilization”) 

-  Muiti-ievel  (individuai  and  group)  and  muiti-functional  iearning 

•  However,  even  Takeuchi  and  Nonaka  noted  that  this  holistic  approach 
may  not  work  in  all  situations: 

-  It  requires  extraordinary  effort  throughout  the  span  of  the  development 
process  (excessive  monthly  overtime  -  not  sustainabie  in  the  long  run) 

-  It  may  not  appiy  to  breakthrough  projects  that  require  innovation 

•  Note  that  immature  technology  belongs  to  this  category 

-  It  may  not  appiy  to  “mammoth”  projects  where  the  sheer  project  scale 
limits  extensive  face-to-face  discussions 

•  Takeuchi  &  Nonaka  specifically  mention  aerospace 

-  The  cuiturai  dimensions  were  not  anaiyzed  for  these  approaches 

•  All  of  the  studied  companies  were  Japanese 
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What  is  Really  Important?  Mission  Success! 


•  There  is  no  declared,  explicit  value  statement  about  Mission  Success 
for  Agile  Development 

-  However,  Mission  Success  in  defense  acquisition  is  essentiai! 

•  The  definition  of  Mission  Success  [Guarro  2007] 

-  The  achievement  by  an  acquired  system  (or  system  of  systems)  to 
singularly  or  in  combination  meet  not  only  specified  performance 
requirements  but  also  expectations  of  users  and  operators  in  terms  of 
safety,  operability,  suitability,  and  supportability 

•  The  definition  of  Mission  Assurance  [Guarro  2007] 

-  The  disciplined  application  of  general  systems  engineering,  quality,  and 
management  principles  towards  the  goal  of  achieving  Mission  Success, 
and  towards  this  goal,  this  disciplined  application  provides  confidence  in 
its  achievement 

•  How  can  you  ensure  that  high  mission  assurance  processes  are  used 
to  develop  your  software? 

-  Use  a  robust  software  development  standard  [Esiinger  06] 

•  Esiinger  argues  that  even  the  use  of  so-called  mature  processes,  such 
as  defined  by  the  CMMI®  is  inadequate,  and  the  government  must 
make  a  robust  software  standard  contractually  compliant 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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Major  Areas  in  a  Typical  Software  Development  Standard* 


System  and  Software  Architecture 

Human  Systems  Integration 

Interoperability  and  Standardization 

Metrics 

Reliability,  Safety,  Information  Assurance 

Project  Planning  and  Oversight 

SW  Development  Environment 

System  Requirements  Analysis 

SW  Requirements  Analysis 

SW  Design 

SW  Implementation  and  Unit  Testing 

Unit  Integration  and  Testing 

SW  Qualification  Testing 

Preparing  for  Transition  to  Operations 
and  Maintenance 

Software  Configuration  Management 

SW  Peer  Reviews  and  Product 
Evaluations 

SW  Quality  Assurance 

Corrective  Action 

Joint  Technical  and  Management 
Reviews 

Risk  Management 

SW  Management  Indicators  (Metrics) 
Security  and  Privacy 
Subcontractor  Management 
Interface  with  Software  IV&V  Agents 


*  Source:  [Adams  05] 
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In  Summary,  What  Can  We  Learn  From  My  Dentist? 


Sign  in  my  dentist’s  office: 

“Brush  only  those  teeth  you  wish  to  keep...” 
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The  Air  Force’s  Agile  Acquisition  Initiative 


•  Championed  by  Dr.  Marvin  R.  Sambur,  then  Assistant  Secretary  of 
the  Air  Force  for  Acquisition* 

-  Theme  #1:  Warfighter  developed  requirements  and  “tossed  over  the 
wall”;  acquirers  tried  to  translate  warfighter  needs  to  contract  documents 

•  Recommendation:  Working  together  as  a  team 

-  This  is  a  true  agile  concern,  Impacting  the  Big  “A” Acquisition 
Process 

-  However,  there  are  problems  with  “jointness”  In  JROC 

-  Managing  service  advocacy  conflicts  with  the  idea  of  managing 
the  DOD  as  an  enterprise 

-  Warfighter  requirements  are  always  tactical;  the  strategic 
perspective  is  missing 


The  underlying  problems  run  deep,  and  the 
recommendation  did  not  offer  tangible  solutions 


*  Source:  [Evans  2003] 
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The  Air  Force’s  Agile  Acquisition  Initiative  (Cent.) 


-  Theme  #2:  Focused  Technology  Transfer 

•  Recommendation:  Foster  closer  working  relationship  between  labs  and 
programs  and  realign  high-priority,  limited  resources 

-  This  is  a  true  agile  concern.  However,  the  majority  of  breakthrough 
technologies,  particularly  technologies  used  in  space,  are  developed 
by  contractors  and  commercial  industry,  and  not  the  Air  Force 
Research  Lab  (AFRL)* 

Labs  can  be  directed  to  work  on  defense  technology 
development,  but  industry  needs  to  be  incentivized 

-  Theme  #3:  “Seams”  between  developmental  testing  and  operational  testing 

•  Recommendation:  Seamless  verification 

-  Bringing-in  testers  early  to  get  advice  on  testability  of  requirements  is 
not  necessarily  an  agile  concern,  but  there  is  no  controversy  involved 

-  However,  it  is  also  a  euphemism  for  incremental  sell-off  of 
requirements 

Impartial  and  independent  Operational  Testing  and 
Evaluation  (OT&E)  must  not  be  weakened 


*  This  is  particularly  true  for  software 
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Defense  Acquisition  Performance  Assessment  of  2006 


•  Selected  DAPA  recommendations  with  possible  agile  consequences* 

-  Targeting  the  PPBE  process  (Budgeting) 

•  Transform  and  stabilize  the  Planning,  Programming,  Budgeting,  and 
Execution  (PPBE)  process 

-  Estabiish  a  distinct,  Stabie  Program  Funding  Account 

-  Program  aii  accounts  to  an  80/20  confidence  ievei 

-  Targeting  the  JCIDS  process  (Requirements) 

•  Replace  the  JCIDS  (Joint  Capabilities  Integration  Development  System) 
with  a  new,  two-year  recurring  planning  process  based  on  the  15-year 
extended  plans  submitted  by  Combatant  Commands 

•  Create  a  new,  “Operationally  Acceptable”  test  category 

-  Targeting  the  iittie  “a” Acquisition  Process 

•  Establish  Time  Certain  Development  as  a  preferred  acquisition  strategy 

-  Make  time  a  Key  Performance  Parameter  (KPP) 

•  Establish  a  realistic  capability  delivery  rate  even  before  MS  A 

•  Complete  the  Test  &  Evaluation  Management  Plan  (TEMP)  and  the 
Initial  Operational  Testing  &  Evaluation  Plan  (lOT&EP)  prior  Milestone  B 

*  Source:  [DAPA  2006] 
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Analysis:  DAPA  and  Agility 


•  Do  DAPA  recommendations  call  for  agility? 

-  The  stabilization  of  the  PPBE  process  is  recommended 

-  The  proposed,  new  requirements  process  wouid  have  a  2-year  duration 

-  The  creation  of  a  stable  Program  Funding  Account  is  recommended 

-  Predictability  is  emphasized,  time  wouid  become  a  KPP 

-  All  accounts  to  be  programmed  to  a  high,  80/20  confidence  level 

-  Realistic  capability  delivery  rate  to  be  established  very  early 

-  Test  plans  to  be  established  very  early 

Most  DAPA  recommendations  do  not  need  agility,  in  fact  they 
emphasize  the  opposite,  i.e.,  stability  and  predictability 
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New  DOD  5000.02  (December  2,  2008) 


Pre-Systems  Acquisition 

System 

Development  & 
Demonstration 
Approval 


Technology 

Development 

Approval 


Milestones: 


Concept 

Refinement 


DOD  5000.02  (May  12,  2003) 
Systems  Acquisition 


Design 

Readiness 

Review 


System  Development 
and 

Demonstration 


Concept 

Decision 


SRR^  SFR^  PDRi 


Low-Rate 
Initial  Prod 
Approval 

^9^ 


IOC 


CDR^  TRR^  SVR^ 


Pre-Systems  Acquisition 


Technology 

Development 

Approval 


Milestones: 


Materiel 

Solution 

Analysis 


Materiel 

Development 

Decision 


Technology 

Development 


Engineering  and 
Manufacturing 
Development 
Approval 


DOD  5000.02  (December  2,  2008) 
Systems  Acquisition 


Post-CDR 

Assessment 


Low-Rate 
Initial  Prod 
Approval 


IOC 


I 


Engineering  and 
Manufacturing 
Development 


SRR^  SFR^  PDR^  CDR^  TRR^  SVR^ 


•  Milestone  B  decision  is  realigned  to  occur  at  the  Preliminary 
Design  Review  (PDR) 

This  alignment  makes  the  front-end  of  the  acquisition  life  cycle 
more  loaded,  indicating  a  waterfall,  rather  than  agile  process 
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Weapon  Systems  Acquisition  Reform  Act 
(WSARA)  of  2009 


DOD  5000.02  (2  December  2008) 


Pre-Systems  Acquisition 

Technology 

Development 

Approval 


Engineering  and 
Manufacturing 
Development 
Approval 


Systems  Acquisition 


Post-CDR 

Assessment 


Low-Rate 
Initial  Prod 
Approval 


Milestones^^^^ 


Materiel 

Engineering  and 

Production 

Solution 

Analysis 

icv«i  II  iv^iv^gy 

Development 

Manufacturing 

Development 

and 

Deployment 

■^Materiel  SRR^  SFR  ^  PDR^  CDR^  TRR'^^  S)//?^ 


Development 

Decision 


DDR&E-conducted  TRAs 
(Annually,  by  March  1st) 


•  The  Director  of  Defense  Research  and  Engineering  (DDR&E) 
shall  annually  conduct  Technology  Readiness  Assessments 
(TRAs)  and  report  results  to  the  DOD  and  Congress 

-  This  is  a  new  requirement,  in  addition  to  the  aiready 
established,  formal  TRAs  at  the  Milestone  Reviews 

Both  the  visibility  and  frequency  of  these  new  reviews  guarantee 
the  slow-down  rather  than  the  speed-up  of  the  process 
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National  Research  Council  /Air  Force  Studies  2009 


•  The  National  Research  Council  (NRC)  was  chartered  to  make 
recommendations  to  streamline  Air  Force  and  DOD  Reviews* 

-  The  committee  identified  31  formai  reviews  of  four  types  that  take  a 
substantiai  toii  on  the  Program  Manager  and  the  Program 
Leadership  Team,  due  to 

•  Excessive  preparation  and  participation  time 

•  Diversion  of  attention  from  the  execution  of  the  program 

•  Significant  cost  imposed 

-  The  recommendations  couid  oniy  suggest  the  eiimination  or 
consoiidation  of  six  reviews 

•  However,  the  list  did  not  show  every  possible  ad-hoc  review 
and  did  not  indicate  the  pre-reviews  and  pre-briefs  generated 
by  these  formal  reviews  either 

This  elaborate  review  structure  and  the  underlying, 
fundamentally  phase-gated  process  inherently  prevents  agility 


*  Source:  [NRC  2009] 
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The  State-of-the-Affairs 


•  Agile  Software  Development  experimentation  is  happening  but 
due  to  numerous  concerns  the  approach  seems  to  be  inadequate 
and  undesirable  in  defense  acquisition 

•  The  Air  Force  Agile  Acquisition  Initiative  did  not  gain  any  traction 
but  was  not  comprehensive  enough  anyway 

•  The  DAPA  report  emphasized  predictability  and  stability:  neither 
of  them  is  an  agile  value 

•  The  current,  big  “A”  acquisition  system  has  serious  counter-lean 
tendencies 

-  Every  new  problem  or  failed  project  triggers  a  proliferation  of 
new  committees,  oversight  boards,  policies,  extensive 
documentation  requirements,  and  processes 

-  This  is  a  systemic  issue  that  needs  to  be  addressed  first 

•  The  NRC  experience  shows  that  streamlining  processes 
is  extremely  difficult  in  the  current  environment 


Peter  Hantos  -  SSTC  2010 


Slide  28 


0 AEROSPACE 


The  Way  Forward 


•  Declaration  of  a  proper  set  of  agile  values  is  needed  for 
defense  acquisition 

-  Agility  is  a  business  strategy,  a  value  proposition  based  on  a  value 
system.  It  is  essential  to  have  a  declared  set  of  values  that  is 

•  Widely  accepted  by  all  stakeholders 

•  Clear  and  unambiguous 

-  However,  it  was  shown  that  different  stakeholders  have 
drastically  different  value  systems  which  need  to  be 
reconciled 

•  Agile  Software  Development  only  uses  one  set  of 
“politically  correct”  values  and  assumes  unconditional 
buy-in  from  all  stakeholders.  Unfortunately,  this 
assumption  is  not  realistic,  even  in  development 

-  While  the  actual  Agile  Software  Development  value  statements 
might  be  inadequate,  the  path  the  authors  of  the  Agile  Manifesto 
devised  to  convey  their  ideas  could  be  proven  useful: 

•  Agile  Values  ->  Agile  Principles  ->  Agile  Practices 
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The  Way  Forward  (cont.) 


•  It  is  possible  that  even  if  a  shared  set  of  values  and  principles 
are  defined  -  depending  on  the  scope  -  multiple  practice 
mappings  are  needed 

-  Despite  what  some  of  the  Agile  Software  Development 
proponents  claim,  agile  software  methods  do  not  scale  well 

-  It  can  be  safely  assumed  that  in  the  localized  contexts  of  J ROC, 

PPBE,  or  the  little  “a”  acquisition  process,  different  agile 
practices  need  to  be  defined 

•  Should  we  emulate  the  business  approach  of  IT  agility 
proposed  by  Drs.  Masini  and  Sengupta  [Masini  2005]? 

-  Their  choice  of  terminology,  i.e.,  time-agility  and  range-agility 
is  not  beneficial  because  it  conflates  the  issues;  Separate  terms 
and  basic  definitions  should  be  established  for  true  agile  values 
and  rapid  execution  values 

•  Of  course  researching  their  relationship  is  still  desirable 

-  Another  caveat  when  using  commercial  models  and  metrics  for 
defense  acquisition  is  that  the  definition  and  role  of  Return  on 
Investment  (ROI)  is  drastically  different  in  the  two  domains 
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The  Way  Forward  (cont.) 


•  The  DAPA  recommendation  for  using  Systems  Dynamics 
analysis  ([DAPA  2006],  Appendix  D,  pp  69-89)  should  be 
rigorously  implemented 

-  The  impact  and  unintended  consequences  of  proposed  new 
practices  should  be  always  evaluated  via  modeling  and  simulation 
before  policy  statements  or  other  guidance  is  drafted 

•  Current,  relevant  research  efforts  at  The  Aerospace  Corporation 

-  Continuous  monitoring  of  acquisition  policy  changes  and 
evaluation  of  cost  and  risk  implications  [Hantos-Kern  2009] 

-  Application  of  Systems  Dynamics  simulation  to  acquisition  and 
development  [Houston-Hantos  2010] 

-  Application  of  Unified  Life  Cycle  Modeling  (ULCM  ®,)  a  modern  life 
cycle  modeling  approach  developed  at  The  Aerospace 
Corporation,  to  explore  concurrent  engineering  risks  in  software¬ 
intensive  systems  development  [Hantos  2008] 

®  ULCM  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  The  Aerospace  Corporation 
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Acronyms 


AFRL 

Air  Force  Research  Laboratory 

CDR 

Critical  Design  Review 

CIVM 

Capability  Maturity  Model  Integration 

DAPA 

Defense  Acquisition  Performance  Assessment 

DDR&E 

Director,  Defense  Research  &  Engineering 

IOC 

Initial  Operational  Capability 

lOT&EP 

Initial  Operational  Testing  &  Evaluation  Plan 

IR&D 

Independent  Research  &  Development 

IV&V 

Independent  Verification  &  Validation 

jaos 

Joint  Capability  Integration  Development  System 

JROC 

Joint  Requirements  Oversight  Council 

KD6I 

Thousand  Delivered  Source  Instructions 

KPP 

Key  Performance  Parameter 

MS 

Milestone 

NRC 

National  Research  Council 

OT&E 

Operational  Test  &  Evaluation 

PDR 

Preliminary  Design  Review 

PPBE 

Planning,  Programming,  Budgeting,  and  Execution 

ROI 

Return  on  Investment 

SFR 

System  Functional  Review 

SRR 

System  Requirements  Review 

SVR 

System  Verification  Review 

SVM 

Software 

TC 

Transformational  Communications 

TBVP 

Test  &  Evaluation  Management  Plan 

TRA 

Technology  Readiness  Assessment 

TRR 

Test  Readiness  Review 

ULCM 

Unified  Life  Cycle  Modeling 

W5ARA 

Weapon  Systems  Acquisition  Reform  Act 
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